[https://nvbugs/6627795][fix] stop charging retiring requests against ADP admission and capacity - #18457

1. What this PR changes

Under the overlap scheduler a request that has emitted its last token is not torn down until the
next iteration. During that window it is retiring: no scheduler will ever forward it again,
but it was still charged against attention-DP admission. Each rank therefore held admission open
for requests that could never be scheduled, offered load could not fill the admission window, and
the context worker ran at roughly half its configured batch. Filed as nvbugs 6627795, 6692514,
6695518, 6704146.

Three modules change.

Attention-DP request routing. Retiring requests are excluded from the per-rank load and token
counts that admission is balanced on, at the single point where rank states are gathered, so all
three routers are corrected at once and none of them needs to know that overlap exists. They are
still counted for liveness and idle-wait decisions, because they remain resident: the liveness
count selects a blocking versus a non-blocking queue wait, and a rank that blocks while its peers
enter a collective hangs rather than slows down. Keeping those two counts distinct is the whole
subtlety here.

Sequence-slot pool sizing — in executor resource sizing, the KV-cache manager (V2) and the
speculative-decoding resource managers. The number of simultaneously-live sequences was being
re-derived from the batch size independently in several places, with formulas that disagreed.
It now has one definition, which is delivered to every pool that indexes by sequence slot: the
V2 index pool, the sampler, the guided decoder and the speculative-decoding slot pools. This half
is what makes the first half real — recovering admission alone is a no-op, because the extra
admitted requests have no slot to occupy and are deferred straight back. The one-iteration
teardown headroom is also generalized from disaggregated-only to any non-PP attention-DP
deployment with overlap enabled, which is what the aggregated case below exercises.

Startup validation. The seat pool and each manager's admissible-sequence count are now
checked for agreement in both directions during initialization, so either direction of skew
fails at startup naming both numbers instead of surfacing as a throughput loss (pool too small)
or as a mid-collective crash (pool too large). A one-sided check is what let this bug through.

Deliberately gated off: pipeline parallelism (it multiplies both sides of the inequality, so the
widening cannot bind), hybrid/SSM architectures (their state pool is sized independently, so an
extra seat would have no state slot behind it), and KV-cache manager V1 (being deprecated — which
is also why the index-pool fix went to V2, the default for the affected models).

2. Perf verification — main tot vs tot + this PR

Two measurement campaigns, kept separate on purpose. The five cases where the
mechanism binds were re-measured on the current commit pair after the PR was
updated; the six inert cases and the overlap-OFF reference arm come from the earlier
pair and are labelled as such. Absolute throughputs are not comparable between the
two campaigns (different wheel build flags), so only within-campaign ratios are quoted.

The head advanced to 34538050da while campaign A was in flight. Stated precisely:
the FIX arm binaries were built at c3f11a2ee0, and 34538050da is a fast-forward adding
one commit that is a proven runtime no-op for all five cases — see below — so the
numbers describe the current head. 21dc97fbc8 is not reused as a base because
63d217f252 is the current merge-base and the two differ by real drift.

Both arms install one byte-identical wheel built from BASE and differ only by which
13 .py files land in site-packages; the overlay is counted per node on every rep
(FIX replaced=13, BASE replaced=0, on every node of every rep). Primary metric
total_token_throughput, median across reps. Campaign A is 30 runs (5 cases × 2 arms ×
3 reps); campaign B contributed 109 runs over 27 case-arm combinations, of which the 60
inert-case runs and the 9 overlap-OFF runs are quoted below. GB300 / GB200 / B200.

The overlap scheduler was confirmed ENABLED on both arms for every rep, on two
independent channels: the resolved context-worker config (disable_overlap_scheduler: false) and the worker's own runtime report. Worth flagging for anyone reproducing this —
the in-repo perf-sanity README states that ctx_only forces disable_overlap_scheduler = True; that is not true at this commit, and had it been, the aggregated row would
have been overlap-OFF and meaningless.

On the PR update itself. The head moved c3f11a2e -> 34538050da during the
campaign. 34538050da re-gates the overlap headroom from the requested KV-cache
manager version to the resolved one, so it changes behaviour only when V2 is requested
and (max_beam_width > 1 or a KV connector is configured). Every case here
reports max_beam_width=1, kv_connector_config=None, use_kv_cache_manager_v2=True
in the worker log, which makes the new expression reduce to the old one exactly — so
these numbers hold for the current head. Cases with beam search or a KV connector are
precisely the ones that commit changes, and are not represented here.

Cases where the mechanism binds — campaign A, current head

Mechanism — two independent observables

Both come from the context worker's own log, not from the throughput metric rescaled,
so they can corroborate or refute the story.

(a) Attention-DP load balance — the cleanest result in the campaign. The context
worker logs currank_total_requests = <this rank>/<all ranks>. Only global rank 0 emits
it, but the denominator is the global total, so rank 0's share is directly computable, and
a balanced role puts it at 1/dp.

Let k be the number of context ranks holding a retiring request at the moment rank
states are gathered. Those ranks over-report their load; the router equalizes apparent
load; and if the over-report is a factor of two the shares are forced to

• k = 0 — nothing penalized, every rank gets 1/dp;
• k ≥ 1 — a penalized rank gets 1/(2·dp − k), an unpenalized one 2/(2·dp − k).

That is a parameter-free ladder of simple rationals — no fitted constants — and it predicts
a discrete set of allowed shares. All 75 reps of the five binding cases land on a rung,
69 of them within 0.10 pp (61 within 0.03 pp). The exceptions are not scattered: they are
the six BASE reps of the glm-5 1k1k case, every one of which sits +0.45 to +0.53 pp
above the 1/3 rung — a systematic offset in one case, not noise, and still nearer that rung
than any other by 16 pp. That case is also the one with the largest context batch (16 seats,
against 2 for the other disagg cases), so the "over-report is a factor of two" idealization is
weakest exactly where the residual appears; the rung is still the right one, but the derivation
is approximate there. Global denominators are identical across arms, so nothing is lost or
added — this is pure redistribution.

The generation role is a useful specificity check. It is also attention-DP (dep8–dep32)
and logged the same way, but shows no quantized structure at all: its rank-0 share
scatters continuously between 1.00× and 1.14× of 1/dp_gen (BASE mean 1.08×, FIX 1.02×).
That is expected — one retiring request is ~50% of the context role's two-seat batch but well
under 1% of a generation rank's, so it cannot move the router by a whole rung. The ladder is a
property of the small-batch role, which is the role this PR's admission change targets.

The rung tells you the state, and the state predicts the throughput:

FIX and OVLOFF are k=0 in 41 of 41 reps. BASE is k=0 in 3 of 34.

Two readings of that table matter, and they pull in opposite directions:

The 28.6% rung is not an escape — it is the same failure seen from a lucky rank. On
dep4, rank 0 starved (14.3%) and rank 0 healthy (28.6%) give 89.89% and 89.26% of
the FIX median — 0.63 pp apart, against a ~±2% noise floor, i.e. indistinguishable. The
genuine k=0 reps sit at 99.98%, 10.1 pp above both. So a run in which rank 0 reports
a comfortable share is still a degraded run; what varies is which rank we can see, not
whether a rank was starved.
Occupancy cannot tell these two apart (rank 0 looks healthy in both), which is why the
28.6% value — not the occupancy trace — is the reliable discriminator.

The 1/dp rung, however, is a real escape, and the previous revision of this section
was right to call BASE bimodal. Three BASE reps routed perfectly uniformly and their
throughput matched their campaign's FIX median to −0.97% / +1.26% / −0.33% — mean
99.98%, i.e. within a tenth of a percent of the fixed arm. Both readings are needed: one
narrows the previous claim, the other confirms it.

The GB200 and GB300 deepseek-r1 rows are identical to the request (2928/20481,
5853/20481) on different clusters, and reps of the same arm and state agree to ±1
request, so each state is deterministic rather than a scheduling draw.

The ladder also predicts the effect size from dp alone. If a starved rank loses one
of its max_num_sequences seats, role capacity goes as dp − k/2, so a k=1 BASE arm
should run at 1 − 1/(2·dp) of FIX:

It over-predicts the loss by 2–4 pp in both — expected, since the context role is not the
only stage in the pipeline — but the ratio is the point: it predicts the dep2 cases
should show 2.33× the dep4 cases' gain, and they show 2.38× (+27.3% vs +11.5%). The
two effect magnitudes in the results table are therefore not two independent measurements;
they are one mechanism read at two DP widths.

The overlap-OFF arm turns this into a causal test. The retiring window exists only
under the overlap scheduler, so if the imbalance is that interaction, base code with
overlap disabled must route balanced. It does — and FIX then reproduces the overlap-OFF
distribution exactly:

Every cell is all three reps of that arm. Across 27 reps the widest within-arm spread is
one request.

Three conclusions, in increasing strength:

• The admission accounting is correct without overlap. With overlap off there is no
  retiring window, so k ≡ 0 by construction and the ladder collapses to 1/dp — which is
  what OVLOFF measures, in 9 of 9 reps. This is an overlap interaction, not a pre-existing
  router defect, which is why it survived review.
• FIX equals OVLOFF to the exact integer, in all three cases and all three reps
  (2563/5121, 5122/10241, 3842/15361 — the same request count, not a close ratio).
  "Recovery is complete, not partial" is usually a claim about percentages inside a noise
  floor; here it is an identity, and the residual FIX-vs-OVLOFF throughput deltas
  (+0.09% / +2.10% / +3.72%) are overlap's own benefit on top of a routing distribution
  that is already bit-identical.
• It is arm-attributable and pair-independent: two cases were measured on both
  commit pairs and come out bit-identical — the glm-5 8k1k case at base 3415/10241 and
  fix 5122/10241, and the aggregated case at base 2197/15361 and fix
  3842/15361. A routing distribution that reproduces to the exact request across two
  different base commits independently corroborates that 34538050da changed nothing for
  these configurations.

This observable is independent of throughput, latency, and occupancy; it is deterministic
rather than distributional; and it measures the mechanism of §1 directly rather than
inferring it. The overlap-OFF cells are from the earlier commit pair (that arm was not
re-run); the BASE/FIX cells reproduce on both pairs.

(b) Context occupancy, on rank 0. num_scheduled_requests per iteration:

The glm-5 8k1k row is worth pausing on: max_batch_size = 2, and BASE sustains exactly
1.000 — one of the two seats is permanently consumed by a retiring request, so the context
role runs at half batch for the entire benchmark, and needs 2.66× the iterations for the
same work. It is the cleanest single-number statement of the bug in the campaign. Both
figures — 6,832 and 2,566 — reproduce to the exact integer on the earlier commit pair as
well, which is a third independent check that the added commit changed nothing here.

These are rank-0 figures, so read them as the occupancy deficit on whichever rank we can
see, not as a role-wide mean; the harness logs iteration stats from rank 0 only, and a
role-wide average is not recoverable from these runs. The 1.996 entry is precisely the rep
whose routing share was 28.578% — rank 0 was unstarved, so its own occupancy looks healthy
while the run is 10.6% down. That is why (a) is the load-bearing observable and this table
is corroboration: occupancy on one rank cannot distinguish "the role is fine" from "some
other rank is starved", and the routing share can.

On the glm-5 1k1k case BASE had 33 index leases available and still sustained only 15
concurrent sequences, so lease supply was never the ceiling — the admission accounting
was.

Refining the previous revision's "bimodality"

The previous revision of this section described BASE as bimodal — a "trapped" mode and an
"escaped" mode whose throughput was "indistinguishable from FIX" — and concluded that "each
delta depends on how often the BASE arm escaped." Having now measured the routing counter,
the escape is confirmed and one part of the claim is narrowed.

Confirmed. The escape is the k=0 rung: 3 BASE reps out of 34 routed perfectly
uniformly, and their throughput matched their campaign's FIX median to −0.97% / +1.26% /
−0.33%. The previously quoted figures reproduce to the digit — the GB300 escape
(105,992.64 against a FIX median of 106,344.60) lands 0.33% below, and the mean of the two
GB200 escapes (94,250.59 and 96,375.19 against 95,178.43) 0.14% above. Equal routing
gives equal throughput on both arms, which remains the signature of an admission fix rather
than a per-iteration speedup.

Narrowed, in three ways.

• The escape was detected by occupancy, and occupancy over-counts it. A rank-0 occupancy
  trace reading "full batch, ~50% of polls idle" is produced both by a genuine k=0 rep
  and by a k=1 rep in which rank 0 happens to be an unstarved rank. The latter is a
  degraded run (89.26% of FIX). Two reps in this campaign are that case, and under the old
  occupancy definition both would have been logged as escapes. The routing share separates
  them cleanly — 25.0% versus 28.6%, each reproducing to ±0.01 pp.
• The escape rate is strongly case-dependent, not a property of the bug. It appears only
  on the two disaggregated deepseek-r1 dep4 cases (2/8 and 1/8 BASE reps) and never on
  the other three — 0 escapes in 18 BASE reps of glm-5 1k1k, glm-5 8k1k and the aggregated
  case. So "each delta depends on how often BASE escaped" holds for the two deepseek-r1
  rows and is not true of the other three, whose BASE arms degrade in every rep. Their
  within-arm spreads run 0.5–3.2% per case-campaign cell, but that is scatter within one
  degraded rung, not a second mode: every one of the 18 reps classifies onto a k ≥ 1 rung.
• Campaign A drew no escapes at all (0 of 15 BASE reps), so every delta in the results
  table above is a degraded-BASE-versus-FIX comparison, not a median straddling two modes.
  That makes these five numbers more stable than campaign B's, not less — but it also
  means they are lower bounds on the spread a reviewer re-running the deepseek-r1 cases
  should expect.

The claim that survives all of this is the previous revision's own final sentence, and it is
worth restating because the ladder now gives it a mechanism: this PR does not make a fast
thing faster, it removes a state that BASE enters in 31 of 34 reps and which costs
10–28% when entered, the exact cost being set by dp.

Cases that cannot show the effect — campaign B, earlier pair

Retained because they establish the noise floor and bound the change's blast radius.
Not re-measured; the mechanism is structurally absent in each, so a newer pair could not
change the conclusion.

The four aggregated rows are token-capped, predicted before the metrics were read:
ctx.max_num_tokens // isl = 8192 // ~7400 = 1, so the context batch admits one request per
iteration regardless of how many leases or seats exist, and no lease or seat change can help.
The con8 row's +3.36% is not a throughput win — its median is flat and the movement is
entirely in the tail (P99 −7.7%), riding on the noisiest BASE arm in the campaign (spread
5.75% vs FIX's 0.95%).

The two gpt-oss rows are the campaign's negative controls, and their inertness is
established affirmatively rather than from an absent log line — five parts: (1) the patch is
verifiably installed, replaced=12 with all three marker symbols present on both nodes
(these rows are campaign B, whose overlay was 12 files — the head move that added a 13th
came later, and it is the very commit these rows control for);
(2) use_kv_cache_manager_v2: false on ctx and gen, both arms; (3) zero V2 index-pool
banners across all 10 reps of each case, so V2 is never constructed; (4) both changed gates
require V2; and (5) enable_attention_dp: false explicitly — which matters because the old
and new headroom gates are not complements, so a hypothetical V1 + disagg + attention-DP
deployment is the one shape where this PR could remove headroom base had. That shape does
not occur here, so the PR cannot make this path worse either.

Those two cases therefore measure environment rather than code, which makes them the
campaign's calibration: the noise floor is ~±2%, with overlapping rep ranges and
single-arm spreads of 2.1–3.4%. Two consequences worth carrying: nothing below ~±2% here
is attributable to this PR, which reclassifies all four aggregated rows as confirmed nulls;
and range non-overlap is not a sufficient separability test at n=3. Both controls read
slightly negative; given the five-part gating the only conceivable channel is import-time
cost, which does not touch steady-state throughput, so this is recorded as an observed
residual rather than a claim that the V1 effect is exactly zero.

They are also the relevant negative control for the newly added commit 34538050da: the
headroom it re-gates is conditioned on the V2 manager, so a V1 case must be untouched, and
both rows sit inside the noise floor.

Caveats, stated rather than smoothed

• 6704147 is still not addressed by this PR. Its context worker is tp1, resolves to
  KV-cache manager V1 and logs enable_attention_dp=False, so every mechanism above is gated
  off. Its regression is real but token-budget-bound rather than seat-bound; it needs the
  non-ADP path extended separately.
• A previously reported "independent variance axis" is now explained, and it is this same
  bug. The earlier revision flagged two GB200 BASE reps that were occupancy-identical on
  the same nodelist yet 12.7% apart, one needing 16.6% more context iterations, and
  recorded it as a variance source this PR does not touch. Both numbers fall out of the
  ladder: those reps are k=2 and k=1, and 3416/2929 = 1.166 reproduces the 16.6%
  exactly while the capacity model predicts (4−1)/(4−0.5) = 1.167. Rank-0 occupancy cannot
  see the difference because rank 0 is starved in both; the routing share can (16.68% vs
  14.30%). That caveat is therefore withdrawn — the excursion is in scope and this PR
  removes it.
• The deepseek-r1 rows are the ones to re-measure if you distrust a median. Their BASE
  arms are the only ones observed to escape (3 of 16 reps), so a BASE median there is a
  median over a discrete mixture rather than over noise. Campaign B's 24.73% BASE spread on
  the GB200 row is entirely this: its five reps were drawn from three different rungs
  (k=0,1,2). Campaign A drew k=1 in every rep, which is why its spread is 1.93%.
• Occupancy figures in table (b) are rank-0-scoped, not role-wide; the harness logs
  iteration stats from rank 0 only.
• Local runs do not upload to OpenSearch, so these rows are not in the perf DB.
• Supersession. The five binding rows supersede the same five rows measured against
  21dc97fbc8, which in turn superseded an n=3 campaign against 3810f4ee50. Absolute
  throughputs moved between campaigns (different wheel build flags), so only within-campaign
  ratios are comparable; the routing counters, being integers, reproduce across all three.

PR Checklist

• Commit message follows [JIRA/NVBUG/None][type] Summary
• Commit signed off (DCO)
• Test cases added for the new behaviour
• pre-commit run clean on all changed files

🤖 Generated with Claude Code